Skip to content

Dont list directorymarkers in iterdir - #536

Open
ap-- wants to merge 3 commits into
fsspec:mainfrom
ap--:dont-list-directorymarkers-in-iterdir
Open

Dont list directorymarkers in iterdir#536
ap-- wants to merge 3 commits into
fsspec:mainfrom
ap--:dont-list-directorymarkers-in-iterdir

Conversation

@ap--

@ap-- ap-- commented Jan 13, 2026

Copy link
Copy Markdown
Collaborator

Should close #226

@ap-- ap-- mentioned this pull request Jan 13, 2026
Open
@danielgafni

danielgafni commented Jan 13, 2026

Copy link
Copy Markdown

Perhaps we should also check for "ContentType": "application/x-directory; charset=UTF-8", as a precaution? This would prevent the special key from filtering out in case the user has actually set it.

@ap--

ap-- commented Jan 13, 2026

Copy link
Copy Markdown
Collaborator Author

It seems, the way info is implemented in s3fs, ContentType is currently not propagated for these directory markers.

@ap--

ap-- commented Jan 13, 2026

Copy link
Copy Markdown
Collaborator Author

If you could explicitly confirm that the PR as-is fixes your issue I'd be happy to merge ☺️

@danielgafni

danielgafni commented Jan 13, 2026 •
edited
Loading

Copy link
Copy Markdown
  1. I don't have access to AWS S3 right now (I can set it up later), so I am testing it with Cloudflare R2. I also artificially created an empty object at s3://anam-data-dev/rofl/ via s3fs I've also created another empty object at s3://anam-data-dev/rofl/object .
  2. I've encountered a different issue now:
/tmp/rofl via 🐍 v3.12.7 (rofl) via   on 🅰 (auto)
› uv pip install git+https://github.com/ap--/universal_pathlib.git@dont-list-directorymarkers-in-iterdir
Resolved 3 packages in 540ms
Uninstalled 1 package in 2ms
░░░░░░░░░░░░░░░░░░░░ [0/1] Installing wheels...                                                                              
Installed 1 package in 6ms
 - universal-pathlib==0.3.8
 + universal-pathlib==0.0.12.post330+gb5d91d25e (from git+https://github.com/ap--/universal_pathlib.git@b5d91d25e2f7acb0e533617a1e38184d375944d9)

/tmp/rofl via 🐍 v3.12.7 (rofl) via   on 🅰 (auto)
› ipython
Python 3.12.7 (main, Oct 16 2024, 04:37:19) [Clang 18.1.8 ]
Type 'copyright', 'credits' or 'license' for more information
IPython 9.9.0 -- An enhanced Interactive Python. Type '?' for help.
Tip: You can find how to type a LaTeX symbol by back-completing it, eg `\θ<tab>` will expand to `\theta`.

In [1]: from upath import UPath

In [2]: path = UPath("s3://anam-data-dev/")

In [3]: list(path.iterdir())
Out[3]:
[S3Path('anam-data-dev/rofl', protocol='s3'),
 S3Path('anam-data-dev/rofl', protocol='s3'),
 S3Path('anam-data-dev/test', protocol='s3'),
 S3Path('anam-data-dev/test.delta', protocol='s3'),
 S3Path('anam-data-dev/video-factory', protocol='s3'),
 S3Path('anam-data-dev/video-generation', protocol='s3')]

In [4]: path = UPath("s3://anam-data-dev/rofl")

In [5]: list(path.iterdir())
---------------------------------------------------------------------------
NotADirectoryError                        Traceback (most recent call last)
Cell In[5], line 1
----> 1 list(path.iterdir())

File /tmp/rofl/.venv/lib/python3.12/site-packages/upath/core.py:1192, in UPath.iterdir(self)
   1190 base_path = base.path
   1191 if not fs.isdir(base_path):
-> 1192     raise NotADirectoryError(str(self))
   1193 name: str
   1194 for info in fs.listdir(base_path):
   1195     # fsspec returns dictionaries

NotADirectoryError: s3://anam-data-dev/rofl

In [6]: path = UPath("s3://anam-data-dev/rofl/")

In [7]: list(path.iterdir())
---------------------------------------------------------------------------
NotADirectoryError                        Traceback (most recent call last)
Cell In[7], line 1
----> 1 list(path.iterdir())

File /tmp/rofl/.venv/lib/python3.12/site-packages/upath/core.py:1192, in UPath.iterdir(self)
   1190 base_path = base.path
   1191 if not fs.isdir(base_path):
-> 1192     raise NotADirectoryError(str(self))
   1193 name: str
   1194 for info in fs.listdir(base_path):
   1195     # fsspec returns dictionaries

NotADirectoryError: s3://anam-data-dev/rofl

So the artificial marker object breaks universal-pathlib right now (for me) when trying to list the marker path. It looks like the requirement here is to have both s3://anam-data-dev/rofl/ and s3://anam-data-dev/rofl/object empty.

@danielgafni

danielgafni commented Jan 13, 2026

Copy link
Copy Markdown

Also this seems a little weird to me

image

@ap--

ap-- commented Jan 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator Author

I don't have access to AWS S3 right now (I can set it up later), so I am testing it with Cloudflare R2. I also artificially created an empty object at s3://anam-data-dev/rofl/ via s3fs I've also created another empty object at s3://anam-data-dev/rofl/object

s3fs doesn't let you do this like the aws console would because it strips the trailing slash

@ap--

ap-- commented Jan 13, 2026

Copy link
Copy Markdown
Collaborator Author

Hmmm, seems like yet another edge case. Okay I'll see if this can be handled in a better way.

@ap-- ap-- added this to the v0.4.0 milestone Jan 31, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

v0.4.0

Development

Successfully merging this pull request may close these issues.

path.iterdir() yields path itself as the first item (with S3)

2 participants

@ap-- @danielgafni